Repository navigation
feat(review): opt-in AI verifier that confirms or refutes each AI comment from dbt evidence - #1769
RaghwendraSingh wants to merge 13 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
There was a problem hiding this comment.
11 issues found across 16 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/opencode/src/altimate/telemetry/index.ts">
<violation number="1" location="packages/opencode/src/altimate/telemetry/index.ts:1625">
P2: The verifier's USD cost is recorded in the envelope but never reaches telemetry. Add an `ai_verify_cost` field here and map it from `env.ai?.verifier?.cost` in review/telemetry.ts, so verifier spend is measurable alongside its tokens.</violation>
</file>
<file name="packages/opencode/src/altimate/review/ai-replay.ts">
<violation number="1" location="packages/opencode/src/altimate/review/ai-replay.ts:121">
P3: Discarded findings are rebuilt from the envelope's `body`, which `applyTriage` clamps to 600 characters. The replayed body is truncated, and the verifier and output see it that way without any `replayIncomplete` flag. Note the truncation in the replay status, or stop clamping for replayed records.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/ai-replay.ts:196">
P1: When replay cannot resolve a verifier model, this branch drops every stored AI finding even though verification never ran. Return `findings` so model-resolution failures fail open like other verifier errors.</violation>
</file>
<file name="packages/opencode/src/altimate/review/format.ts">
<violation number="1" location="packages/opencode/src/altimate/review/format.ts:390">
P2: A configured `ai-review:*: inline` rule can overwrite the verifier’s `summary` tier, and this condition then bypasses `verifierDemoted`. Check verifier demotion unconditionally before allowing inline-tier findings.</violation>
</file>
<file name="packages/opencode/src/altimate/review/git.ts">
<violation number="1" location="packages/opencode/src/altimate/review/git.ts:207">
P3: A valid filename like `models/a:123:b.sql` is split at the filename’s `:123:` and returned with a false path and line. Parse an unambiguous path/line format instead.</violation>
</file>
<file name="packages/opencode/src/altimate/review/ai-verify.ts">
<violation number="1" location="packages/opencode/src/altimate/review/ai-verify.ts:198">
P2: A finding with no usable line and no quoted or backticked identifier is classified as targeting unchanged code, so triage discards it as `unchanged_code` unless the model confirmed it. An empty identifier list should not count as evidence of unchanged code. Require at least one identifier before marking it unchanged.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/ai-verify.ts:1003">
P2: When a batch's share cannot cover even its direct-answer estimate, `force` still sends that call; repeated across batches, verifier usage can exceed the 12k PR budget. Skip that batch as budget-exhausted when the answer itself cannot fit, leaving its findings unreviewed.</violation>
</file>
<file name="packages/opencode/src/altimate/review/runner.ts">
<violation number="1" location="packages/opencode/src/altimate/review/runner.ts:471">
P2: `columnReach` records an expected "model not found" as an engine failure via `fail()`. That failure lands in `engineErrors`, which downgrades an APPROVE to COMMENT and marks the run as not completed. Return `undefined` for not-found without calling `fail`, as `modelConsumers` does, and reserve `fail` for real errors such as an unreadable manifest.</violation>
</file>
<file name="packages/opencode/src/altimate/llm/stream-text.ts">
<violation number="1" location="packages/opencode/src/altimate/llm/stream-text.ts:32">
P3: `readStreamStep` adds a new error-propagation path and an `invalid` passthrough, but no test covers either. Add cases to stream-text.test.ts for a mid-stream `error` part (must throw) and for a tool call with `invalid: true`.</violation>
</file>
<file name="packages/opencode/src/altimate/review/verdict.ts">
<violation number="1" location="packages/opencode/src/altimate/review/verdict.ts:237">
P3: The `dropped` schema accepts any decision and any length, but the producer only records discards and caps the list at 20. Narrow `decision` to `z.literal("discard")` and add `.max(20)` to the array so the schema states the invariant.</violation>
</file>
<file name="packages/opencode/src/altimate/review/ai-review.ts">
<violation number="1" location="packages/opencode/src/altimate/review/ai-review.ts:312">
P2: With the verifier on, lane-level `tokens`, `cost` and `calls` cover only the generator, so `ai_cost`/`ai_tokens_total`/`ai_calls` in telemetry undercount total AI spend. Either add the verifier's usage into these totals or document that they exclude it, and update the `tokens`/`calls` JSDoc to match.</violation>
</file>
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
View guided diff | Turn on auto-fix | Re-trigger cubic
| model = await Provider.getModel(chosen.providerID, chosen.modelID) | ||
| } catch (err) { | ||
| const reason = `no usable model: ${err instanceof Error ? err.message : String(err)}`.slice(0, 200) | ||
| return done([], { ...lane.status, state: "skipped", reason }) |
There was a problem hiding this comment.
P1: When replay cannot resolve a verifier model, this branch drops every stored AI finding even though verification never ran. Return findings so model-resolution failures fail open like other verifier errors.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-replay.ts, line 196:
<comment>When replay cannot resolve a verifier model, this branch drops every stored AI finding even though verification never ran. Return `findings` so model-resolution failures fail open like other verifier errors.</comment>
<file context>
@@ -0,0 +1,203 @@
+ model = await Provider.getModel(chosen.providerID, chosen.modelID)
+ } catch (err) {
+ const reason = `no usable model: ${err instanceof Error ? err.message : String(err)}`.slice(0, 200)
+ return done([], { ...lane.status, state: "skipped", reason })
+ }
+ const v = await runAiVerify({ findings, grounding: input.grounding, evidence: input.evidence, model })
</file context>
| try { | ||
| return columnReach(opts.manifestPath, model, column, dialect) | ||
| } catch (err) { | ||
| fail("columnReach", err) |
There was a problem hiding this comment.
P2: columnReach records an expected "model not found" as an engine failure via fail(). That failure lands in engineErrors, which downgrades an APPROVE to COMMENT and marks the run as not completed. Return undefined for not-found without calling fail, as modelConsumers does, and reserve fail for real errors such as an unreadable manifest.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/runner.ts, line 471:
<comment>`columnReach` records an expected "model not found" as an engine failure via `fail()`. That failure lands in `engineErrors`, which downgrades an APPROVE to COMMENT and marks the run as not completed. Return `undefined` for not-found without calling `fail`, as `modelConsumers` does, and reserve `fail` for real errors such as an unreadable manifest.</comment>
<file context>
@@ -463,6 +464,15 @@ export function createDispatcherRunner(opts: DispatcherRunnerOptions): ReviewRun
+ try {
+ return columnReach(opts.manifestPath, model, column, dialect)
+ } catch (err) {
+ fail("columnReach", err)
+ return undefined
+ }
</file context>
| .filter((l) => l.kind === "+" || added.has(l.line)) | ||
| .map((l) => l.text.toLowerCase()) | ||
| const ids = quotedNames(text) | ||
| targetsUnchangedCode = !ids.some((id) => addedText.some((t) => t.includes(id))) |
There was a problem hiding this comment.
P2: A finding with no usable line and no quoted or backticked identifier is classified as targeting unchanged code, so triage discards it as unchanged_code unless the model confirmed it. An empty identifier list should not count as evidence of unchanged code. Require at least one identifier before marking it unchanged.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-verify.ts, line 198:
<comment>A finding with no usable line and no quoted or backticked identifier is classified as targeting unchanged code, so triage discards it as `unchanged_code` unless the model confirmed it. An empty identifier list should not count as evidence of unchanged code. Require at least one identifier before marking it unchanged.</comment>
<file context>
@@ -0,0 +1,1230 @@
+ .filter((l) => l.kind === "+" || added.has(l.line))
+ .map((l) => l.text.toLowerCase())
+ const ids = quotedNames(text)
+ targetsUnchangedCode = !ids.some((id) => addedText.some((t) => t.includes(id)))
+ }
+
</file context>
| targetsUnchangedCode = !ids.some((id) => addedText.some((t) => t.includes(id))) | |
| targetsUnchangedCode = ids.length > 0 && !ids.some((id) => addedText.some((t) => t.includes(id))) |
| ai_verify_tool_calls?: number | ||
| /** AI findings the verifier kept (including those moved to the summary) and discarded. */ | ||
| ai_verify_kept?: number | ||
| ai_verify_discarded?: number |
There was a problem hiding this comment.
P2: The verifier's USD cost is recorded in the envelope but never reaches telemetry. Add an ai_verify_cost field here and map it from env.ai?.verifier?.cost in review/telemetry.ts, so verifier spend is measurable alongside its tokens.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/telemetry/index.ts, line 1625:
<comment>The verifier's USD cost is recorded in the envelope but never reaches telemetry. Add an `ai_verify_cost` field here and map it from `env.ai?.verifier?.cost` in review/telemetry.ts, so verifier spend is measurable alongside its tokens.</comment>
<file context>
@@ -1613,6 +1613,16 @@ export namespace Telemetry {
+ ai_verify_tool_calls?: number
+ /** AI findings the verifier kept (including those moved to the summary) and discarded. */
+ ai_verify_kept?: number
+ ai_verify_discarded?: number
/** Where the review base came from (flag, github_event, default_branch, …). */
base_source?: string
</file context>
| ai_verify_discarded?: number | |
| ai_verify_discarded?: number | |
| /** Estimated USD cost of the verifier from the model's price table. */ | |
| ai_verify_cost?: number |
| findings = v.findings | ||
| verifier = v.status | ||
| } | ||
| return done(findings, { state: "ok", model: modelLabel, ...(reason ? { reason } : {}), ...spent, ...(verifier ? { verifier } : {}) }) |
There was a problem hiding this comment.
P2: With the verifier on, lane-level tokens, cost and calls cover only the generator, so ai_cost/ai_tokens_total/ai_calls in telemetry undercount total AI spend. Either add the verifier's usage into these totals or document that they exclude it, and update the tokens/calls JSDoc to match.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-review.ts, line 312:
<comment>With the verifier on, lane-level `tokens`, `cost` and `calls` cover only the generator, so `ai_cost`/`ai_tokens_total`/`ai_calls` in telemetry undercount total AI spend. Either add the verifier's usage into these totals or document that they exclude it, and update the `tokens`/`calls` JSDoc to match.</comment>
<file context>
@@ -281,15 +294,22 @@ export async function runAiReview(input: AiReviewInput): Promise<AiReviewResult>
+ findings = v.findings
+ verifier = v.status
+ }
+ return done(findings, { state: "ok", model: modelLabel, ...(reason ? { reason } : {}), ...spent, ...(verifier ? { verifier } : {}) })
} catch (err) {
log.error("ai review failed", { error: err })
</file context>
| const roundOutput = Math.min(VERIFY_MAX_CALLS_PER_ROUND * TOOL_OUTPUT_MAX, ctx.maxToolChars - toolChars) | ||
| const roomForRound = tokens + nextCall() + nextCall(roundOutput) <= ctx.maxTokens | ||
| const overBudget = !roomForRound || toolChars > ctx.maxToolChars | ||
| const force = rounds >= ctx.maxRounds || overBudget |
There was a problem hiding this comment.
P2: When a batch's share cannot cover even its direct-answer estimate, force still sends that call; repeated across batches, verifier usage can exceed the 12k PR budget. Skip that batch as budget-exhausted when the answer itself cannot fit, leaving its findings unreviewed.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-verify.ts, line 1003:
<comment>When a batch's share cannot cover even its direct-answer estimate, `force` still sends that call; repeated across batches, verifier usage can exceed the 12k PR budget. Skip that batch as budget-exhausted when the answer itself cannot fit, leaving its findings unreviewed.</comment>
<file context>
@@ -0,0 +1,1230 @@
+ const roundOutput = Math.min(VERIFY_MAX_CALLS_PER_ROUND * TOOL_OUTPUT_MAX, ctx.maxToolChars - toolChars)
+ const roomForRound = tokens + nextCall() + nextCall(roundOutput) <= ctx.maxTokens
+ const overBudget = !roomForRound || toolChars > ctx.maxToolChars
+ const force = rounds >= ctx.maxRounds || overBudget
+ if (overBudget && rounds < ctx.maxRounds) out.state = "budget"
+ if (force && rounds > 0) messages.push({ role: "user", content: ANSWER_NOW })
</file context>
| * by the SDK; the caller runs them and appends the results. Like | ||
| * `readStreamText`, a stream error is thrown rather than lost. | ||
| */ | ||
| export async function readStreamStep(stream: { |
There was a problem hiding this comment.
P3: readStreamStep adds a new error-propagation path and an invalid passthrough, but no test covers either. Add cases to stream-text.test.ts for a mid-stream error part (must throw) and for a tool call with invalid: true.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/llm/stream-text.ts, line 32:
<comment>`readStreamStep` adds a new error-propagation path and an `invalid` passthrough, but no test covers either. Add cases to stream-text.test.ts for a mid-stream `error` part (must throw) and for a tool call with `invalid: true`.</comment>
<file context>
@@ -12,3 +12,42 @@ export async function readStreamText(stream: {
+ * by the SDK; the caller runs them and appends the results. Like
+ * `readStreamText`, a stream error is thrown rather than lost.
+ */
+export async function readStreamStep(stream: {
+ fullStream: AsyncIterable<{ type: string; error?: unknown }>
+ readonly text: PromiseLike<string>
</file context>
| summary: z.number().int().nonnegative().optional(), | ||
| unreviewed: z.number().int().nonnegative().optional(), | ||
| /** Discarded findings (at most 20), for auditing the verifier's precision. */ | ||
| dropped: z |
There was a problem hiding this comment.
P3: The dropped schema accepts any decision and any length, but the producer only records discards and caps the list at 20. Narrow decision to z.literal("discard") and add .max(20) to the array so the schema states the invariant.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/verdict.ts, line 237:
<comment>The `dropped` schema accepts any decision and any length, but the producer only records discards and caps the list at 20. Narrow `decision` to `z.literal("discard")` and add `.max(20)` to the array so the schema states the invariant.</comment>
<file context>
@@ -201,18 +210,80 @@ export const VerdictEnvelope = z.object({
+ summary: z.number().int().nonnegative().optional(),
+ unreviewed: z.number().int().nonnegative().optional(),
+ /** Discarded findings (at most 20), for auditing the verifier's precision. */
+ dropped: z
+ .array(
+ z.object({
</file context>
| severity: d.severity === "warning" ? "warning" : "suggestion", | ||
| category: category.data, | ||
| title: d.title, | ||
| body: d.body, |
There was a problem hiding this comment.
P3: Discarded findings are rebuilt from the envelope's body, which applyTriage clamps to 600 characters. The replayed body is truncated, and the verifier and output see it that way without any replayIncomplete flag. Note the truncation in the replay status, or stop clamping for replayed records.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-replay.ts, line 121:
<comment>Discarded findings are rebuilt from the envelope's `body`, which `applyTriage` clamps to 600 characters. The replayed body is truncated, and the verifier and output see it that way without any `replayIncomplete` flag. Note the truncation in the replay status, or stop clamping for replayed records.</comment>
<file context>
@@ -0,0 +1,203 @@
+ severity: d.severity === "warning" ? "warning" : "suggestion",
+ category: category.data,
+ title: d.title,
+ body: d.body,
+ file: d.file,
+ model: modelNameFromPath(d.file),
</file context>
| const hits: Array<{ path: string; line: number; text: string }> = [] | ||
| for (const row of out.split("\n")) { | ||
| const rest = ref && row.startsWith(ref + ":") ? row.slice(ref.length + 1) : row | ||
| const m = /^(.+?):(\d+):(.*)$/.exec(rest) |
There was a problem hiding this comment.
P3: A valid filename like models/a:123:b.sql is split at the filename’s :123: and returned with a false path and line. Parse an unambiguous path/line format instead.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/git.ts, line 207:
<comment>A valid filename like `models/a:123:b.sql` is split at the filename’s `:123:` and returned with a false path and line. Parse an unambiguous path/line format instead.</comment>
<file context>
@@ -178,6 +178,40 @@ export async function grepMatchesAt(cwd: string, ref: string | undefined, patter
+ const hits: Array<{ path: string; line: number; text: string }> = []
+ for (const row of out.split("\n")) {
+ const rest = ref && row.startsWith(ref + ":") ? row.slice(ref.length + 1) : row
+ const m = /^(.+?):(\d+):(.*)$/.exec(rest)
+ if (!m) continue
+ hits.push({ path: m[1], line: Number(m[2]), text: m[3].slice(0, 300) })
</file context>
| compiledSql: async (p, side) => { | ||
| const ctx = ctxByPath.get(p) | ||
| const engine = side === "new" ? ctx?.engineNewSql : ctx?.engineOldSql | ||
| return engine ?? (await getCompiled?.(ctx?.file.oldPath && side === "old" ? ctx.file.oldPath : p, side)) |
There was a problem hiding this comment.
WARNING: compiled_sql falls back to on-disk compiled artifacts for a side the PR does not have
engineNewSql/engineOldSql are deliberately undefined for a deleted head or an added base (see lines 1844-1856, which never call getCompiled for status === "deleted" or without an old side). This new closure ignores that: ctx exists for a deleted model with engineNewSql === undefined, so it calls getCompiled(p, "new"), which reads target/compiled/<project>/<path> from disk with no ref or status check (compiled.ts:108-121). dbt compile does not purge stale artifacts, so compiled_sql(path, "new") — and equivalence() built on it — can show the deleted model's old SQL as head, letting the verifier confirm a claim about code that no longer exists while read_file(path, "new") correctly reports it missing.
Return engine when ctxByPath has the path (even when it is undefined) and use getCompiled only for parents/downstream models outside that map.
| const near = (r: { head?: number }, i: number) => { | ||
| if (typeof line !== "number") return i < 2 * HUNK_RADIUS | ||
| // A removed row or hunk header sits beside the head row that follows it. | ||
| const at = r.head ?? rows.slice(i).find((x) => x.head !== undefined)?.head ?? right |
There was a problem hiding this comment.
WARNING: hunkAround anchors removed rows to the next hunk and drops blank context rows
rows.slice(i).find(...) scans the whole accumulated row list, not the current hunk. A hunk that ends in deletions has no following head row of its own, so its - rows anchor to the next hunk's first head line: they are omitted from the excerpt for a finding in that hunk (the deleted code is missing exactly where the finding points) and included in a distant hunk's excerpt as if adjacent. Separately, row === "" (a blank context line that lost its leading space) is skipped entirely, so right is not incremented and every later row in the excerpt is numbered one too low; inHunk is also never reset for unrecognized rows.
walkDiffRows (diff-filter.ts:349-372) already numbers +, context, - and blank rows correctly — reuse it and mark hunk-header rows separately instead of re-deriving the walk here. The verifier decides confirm/refute from this excerpt, so a missing deletion or shifted numbers can refute a valid comment.
| const dirs = layout | ||
| ? [...layout.dirs.model, ...layout.dirs.macro, ...layout.dirs.seed, ...layout.dirs.snapshot, ...layout.dirs.test] | ||
| : ["models", "macros", "seeds", "snapshots", "tests"] | ||
| const specs = glob ? [`${root}${glob}`] : dirs.map((d) => path.posix.normalize(root + d)) |
There was a problem hiding this comment.
WARNING: A repo-relative glob is prefixed with the project root a second time
The verifier is shown repo-relative paths (file: ${f.file} in buildVerifyMessage, and grep_project results are repo-relative), so in a monorepo the model naturally passes glob: "dbt/models/**". root + glob then becomes dbt/dbt/models/**, git matches nothing, and the tool returns the literal "no matches" — which the verifier can read as proof of absence and use to refute a correct cross-file claim.
Normalize the glob, only add projectRootRel when it is not already present, and reject any .. segment (dbt/../secrets/** currently passes the startsWith("..") filter). Keep the resulting spec inside projectRootRel.
| return { decision: "keep", rule: "plausible_local" } | ||
| if (m.verdict === "unverifiable") return { decision: "discard", rule: "unverifiable" } | ||
| if (m.value === "informational") return { decision: "summary", rule: "informational" } | ||
| return { decision: "keep", rule: "verified" } |
There was a problem hiding this comment.
SUGGESTION: A refuted verdict with no cited evidence is recorded as verified
Row 2 only discards a refutation that carries evidence (m.evidence.trim()), so an answer like {"verdict":"refuted","evidence":""} falls through every later rule and reaches this unconditional return. applyTriage then writes verification: { decision: "keep", rule: "verified", verdict: "refuted" } into the signed envelope and into decisions[], which is contradictory audit data for offline re-triage. (No Verified: suffix is appended because that also needs evidence, so this is a labeling defect rather than posted text.)
Make the last row conditional on m.verdict === "confirmed" and use a distinct rule such as refuted_unsupported otherwise.
| return s.length > TOOL_OUTPUT_MAX ? s.slice(0, TOOL_OUTPUT_MAX) + "\n… (truncated)" : s | ||
| } | ||
|
|
||
| function safeReason(err: unknown): string { |
There was a problem hiding this comment.
SUGGESTION: Duplicate helpers instead of reusing the AI lane's
safeReason here is byte-for-byte the same as ai-review.ts:190; positiveNumber duplicates aiInputBudget's num (ai-review.ts:37-40); and BATCH_CONCURRENCY is declared in both modules. Export the existing helpers and import them so key/bearer-token masking and env parsing cannot drift between the two lanes. backticked() in this file is also a strict subset of quotedNames() and can be dropped.
| } | ||
| let body = f.body | ||
| const suffix = verification?.rule === "verified" && verification.evidence ? `\n\nVerified: ${verification.evidence}` : "" | ||
| if (suffix && body.endsWith(suffix)) body = body.slice(0, -suffix.length) |
There was a problem hiding this comment.
SUGGESTION: unverify only strips a Verified: suffix that is the tail of the body, which mergeRepeats breaks
applyTriage appends the suffix, but cleanFindings then runs mergeRepeats (finding.ts:291-295), which rewrites the first finding of a group to ${first.body}\n\nSame rule, also here:…. If a verified finding wins that tie-break, the stored body is …Verified: <evidence>\n\nSame rule, also here:…, so body.endsWith(suffix) is false and replay keeps a Verified: claim the new verifier never made (it is fed f.body by buildVerifyMessage/extractFeatures, and is posted if re-kept). The merged (+N more) title is likewise not undone.
Remove the last occurrence of the suffix instead of requiring a tail match, and strip the (+N more) marker when rebuilding the generator's finding.
Code Review SummaryStatus: 6 Issues Found | Recommendation: Address before merge Incremental review of Overview
Issue Details (click to expand)WARNING
SUGGESTION
Files Reviewed (7 files)
Fix these issues in Kilo Cloud Previous Review Summary (commit 8aba994)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit 8aba994)Status: 6 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)WARNING
SUGGESTION
Files Reviewed (16 files)
Reviewed by deepseek-v4.1-flash-1017 · Input: 176K · Output: 22.3K · Cached: 1.5M Review guidance: REVIEW.md from base branch |
hkaltai
left a comment
There was a problem hiding this comment.
Verdict: approve with minor changes. Panel: Claude, Codex, and a lead pass, all covering every changed file (reviewed against the stack base only). No CRITICAL or MAJOR defects; flag-off behaviour is unchanged and the verifier fails open as described. bun test test/altimate/review passes (667).
| Severity | Count |
|---|---|
| CRITICAL | 0 |
| MAJOR | 0 |
| MINOR | 6 (inline) |
| NIT | 6 |
Consensus (2+ reviewers): config-rule override of summary demotion, replay dropping findings on model failure. Single-reviewer findings were verified against the code. A Codex-rated MAJOR (token budget exceeded) was downgraded to MINOR: it fails safe and degrades to the summary, but the 12k bound is not hard. Missing tests worth adding: grepFixedAt / grepProject wiring, new review_run telemetry fields, and the regressions for the findings above.
Nits
telemetry/index.ts:1623/** AI findings the verifier kept (including those moved to the summary) and discarded. */- summary decisions are counted separately andsummary/unreviewedare not emitted. Fix the comment or emitai_verify_summary/ai_verify_unreviewed.ai-verify.ts:1224log.error("ai verify failed", ...)path returns findings unchanged even withALTIMATE_REVIEW_AI_VERIFY_FAIL=closed. ApplyfailModein the catch or document that closed mode is per-verdict only.ai-verify.ts:369rows.slice(i).find(...)is O(n^2) per candidate on large diffs. Precompute next head line in one backward pass.ai-verify.ts:88,:447-449experiment notes ("opt-in until the decision run says otherwise", "in the decision run about half of these discards were judged valuable") will go stale; move to the PR/design doc.:3("One extra, bounded model call") and:637("may spend before it must answer") overstate a hard bound.ai-replay.ts:78-82"as the generator built them" overstates fidelity: discarded bodies are clamped to 600 chars and kept findings have been through anchor/clean post-processing. Persist raw generator findings or document, and strippreExisting/evidenceLineinunverify.- Naming:
ts/TsFeatures(saved as envelope keydecisions[].ts) reads as timestamp/TypeScript; rename tofeatures.keptvssummaryis ambiguous in status/telemetry.DroppedFinding.decisionis always"discard". Generic exportedDecision/Verificationnames.orchestrate.ts:162-167graphParentsdoc says project-relative but paths are repo-relative.
Also worth noting: verifier tokens/cost are not added to ai.cost/ai.tokens (only under ai.verifier), and ALTIMATE_REVIEW_AI_REPLAY is honoured in production runAiReview.
| * the PR added, at or above the severity floor. */ | ||
| export function inlineEligible(f: Finding, posting: PostingPolicy): boolean { | ||
| if (f.tier !== "inline" && findingOrigin(f) !== "ai") return false | ||
| if (f.tier !== "inline" && (findingOrigin(f) !== "ai" || verifierDemoted(f))) return false |
There was a problem hiding this comment.
[MINOR] Verifier demotion can be overridden by a config rule tier
if (f.tier !== "inline" && (findingOrigin(f) !== "ai" || verifierDemoted(f))) return falseverifierDemoted is only consulted when tier !== "inline", and cleanFindings re-tiers every finding from ruleTier(f.rule, config.rules) after the AI lane, overwriting the summary tier set by triage. With a rule such as ai-review/*: inline in the review config, an unverifiable cross-file finding the verifier moved to the summary becomes inline-eligible again and is posted inline. Reproduced independently by two reviewers.
Fix: check verifierDemoted(f) first (return false), or keep summary in cleanFindings when verification.decision === "summary". Add a regression test.
| model = await Provider.getModel(chosen.providerID, chosen.modelID) | ||
| } catch (err) { | ||
| const reason = `no usable model: ${err instanceof Error ? err.message : String(err)}`.slice(0, 200) | ||
| return done([], { ...lane.status, state: "skipped", reason }) |
There was a problem hiding this comment.
[MINOR] Replay drops all replayed findings when no verifier model resolves
return done([], { ...lane.status, state: "skipped", reason })The generator findings are already rebuilt at this point, but are discarded on a model-lookup failure, contradicting the module's fail-open design. Scenario: ALTIMATE_REVIEW_AI_REPLAY=env.json ALTIMATE_REVIEW_AI_VERIFY=1 with an unavailable or misnamed --ai-model yields zero AI findings, silently skewing an A/B comparison.
Fix: return lane.findings unchanged with verifier: { state: "skipped", reason }, and add a test.
| ? { decision: "keep", rule: "unreviewed" } | ||
| : { decision: "discard", rule: "unreviewed" } | ||
| if (ts.restatesGrounding) return { decision: "discard", rule: "duplicate_of_engine" } | ||
| if (m.verdict === "refuted" && m.evidence.trim()) return { decision: "discard", rule: "refuted" } |
There was a problem hiding this comment.
[MINOR] Refuted verdict without evidence is kept as "verified" while unverifiable is discarded
if (m.verdict === "refuted" && m.evidence.trim()) return { decision: "discard", rule: "refuted" }A refuted verdict with blank evidence skips this row and every unverifiable row, and falls through to keep / "verified". For a finding with evidenceLineValid: false, unverifiable is discarded but refuted with empty evidence is kept and labelled verified, so the model saying the claim is wrong makes it more likely to survive.
Fix: normalise refuted-without-evidence to unverifiable before the remaining rows (or give it its own rule name so it is never labelled verified), and test the invalid-line case.
| Math.ceil((requestChars(ctx.fixedChars, messages) + ANSWER_NOW.length + extra) / charsPerToken) + | ||
| VERIFY_OUTPUT_RESERVE | ||
| const roundOutput = Math.min(VERIFY_MAX_CALLS_PER_ROUND * TOOL_OUTPUT_MAX, ctx.maxToolChars - toolChars) | ||
| const roomForRound = tokens + nextCall() + nextCall(roundOutput) <= ctx.maxTokens |
There was a problem hiding this comment.
[MINOR] Token budget is a planning target, not a bound; tool rounds rarely run
const roomForRound = tokens + nextCall() + nextCall(roundOutput) <= ctx.maxTokensTwo related effects. (1) The forced answer is always sent, even when a batch's minimum request exceeds its budget share, and the output reserve is not enforced; with 40 findings (5 batches) an in-memory harness spent ~26.7k tokens against the 12k default, so the documented 12k bound can be exceeded. (2) roundOutput assumes the worst case (6 x 2,500 chars) and maxTokens is split across batches, so with 8 findings (one batch) the first call is already toolChoice: "none" and with 9 findings both calls are; the tool loop effectively never runs on mid-size PRs and the state reports budget. Fails safe (cross-file claims go to the summary), but limits both the cost bound and the verifier's usefulness.
Fix: plan with a realistic round cost (e.g. TOOL_OUTPUT_MAX * 2) or give each batch a floor allowing one round; skip batches whose minimum request cannot fit (keep findings unchanged); document the cap as soft otherwise.
| if (ctx.abort.aborted) throw ctx.abort.reason ?? new Error("aborted") | ||
| if (force || step.finishReason !== "tool-calls" || !step.toolCalls.length) { | ||
| out.verdicts = parseVerifierOutput(step.text, ids) | ||
| if (!out.verdicts.size && out.state === "ok") { |
There was a problem hiding this comment.
[MINOR] Unparsable forced answer is reported as budget, not error
if (!out.verdicts.size && out.state === "ok") {When the answer was forced early (state: "budget") and the model returns nothing parsable (e.g. []), the batch keeps budget with no error, although every finding is unreviewed. Reproduced with 8 findings and an LLM reply of "[]". Telemetry ai_verify_state then hides verifier failures.
Fix: set error/reason whenever verdicts.size === 0, regardless of the earlier state.
| toolChoice: force ? "none" : "auto", | ||
| abort: ctx.abort, | ||
| } | ||
| const step = await ctx.llm(req) |
There was a problem hiding this comment.
[MINOR] Timeout cannot interrupt awaited evidence tools
const step = await ctx.llm(req)The timeout aborts model requests only; await runToolCall(...) in the same loop (and graph gathering before the timer starts) is not raced against the abort signal. One reviewer reproduced a pending evidence read keeping the verifier unresolved after a 10 ms timeout until the read was released. Local git/file reads rarely stall, so impact is limited, but it weakens the fail-open-on-timeout guarantee.
Fix: apply one deadline across graph gathering, tools and model calls (race pending work against the abort signal and stop scheduling after expiry).
Adds the skeleton of a dbt-grounded verifier for the advisory AI lane, behind `ALTIMATE_REVIEW_AI_VERIFY=1`. With the variable unset nothing changes: the lane gets no `evidence` and the envelope is identical. - `ai-verify.ts`: `AiVerifyEvidence`, `VerifierStatus`, `aiVerifyEnabled` and a `runAiVerify` that returns the findings unchanged for now - `runAiReview` runs the verifier after the review batches when given `evidence` and the variable is set; `AiLaneStatus.verifier` reports it - `runReview` builds the evidence when `aiVerify` is set: engine SQL, numbered file slices, added lines, PR text and a project grep - `run.ts` passes `aiVerify` and a `git grep -F` backed `grepProject` (5 s timeout, 50 hits) over the project's dbt directories - `ReviewRunner.columnReach` wraps `columnReach` without throwing - `readStreamStep` reads one AI SDK step keeping its tool calls and response messages - envelope `ai.verifier` and `review_run` fields `ai_verify_state`, `ai_verify_tokens_total`, `ai_verify_tool_calls`, `ai_verify_kept`, `ai_verify_discarded` Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`extractFeatures` computes, without a model, the facts the verifier's triage table reads for each AI finding: - `evidenceLineValid`: the finding's line is on the diff (null without one) - `targetsUnchangedCode`: a diff line the PR did not add, a `preExisting` tag, or no usable line and nothing the finding names on an added line - `restatesGrounding`: title-word Jaccard of at least 0.5 with an engine finding on the same file, or the same category and column - `claimsOtherFile`: a `ref()` to another model, a `source()`, or a backticked name of a dbt parent or downstream model - `styleOnly`, `lowConfidence`, `hedgedDataDiff` AI findings reach the verifier before `anchorToDiff` tags `preExisting`, so a commentable line outside the added set counts as unchanged code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- `VERIFY_SYSTEM_PROMPT`: an adversarial verifier that tries to refute each candidate from its hunk and tool results, treats the diff, SQL, PR text and tool output as untrusted data, and answers with one JSON element per candidate (`claim_type`, `verdict`, `value`, `requires_assumed_input`, `evidence`, `tools_used`) - `buildVerifyMessage`: the PR title and, per candidate, its id, file, line, category, severity, title, body and the diff rows within 15 lines of it (`hunkAround`, at most 2,000 chars); never the whole file or the engine findings the reviewer saw - `parseVerifierOutput`: strips `<think>` blocks and fences, takes the first balanced JSON array, ignores unknown ids and coerces bad enums (`verdict` to unverifiable, `claim_type` to code_fact, `value` to actionable); ids it does not return stay unreviewed The prompt lives in TypeScript for the decision run and can move to core later. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`triage` decides each AI finding from its deterministic features and the verifier's verdict, first matching row wins: grounding duplicate, refuted with evidence, unverifiable cross-file claim, speculation or an assumed input, style or nit, unchanged code, plausible local claim (kept), unverifiable, informational (summary), verified (kept). Without a verdict the fail mode decides: open keeps the finding unchanged. `applyTriage` records `evidence.result.verification` on reviewed kept findings (the evidence tool stays `ai-review`), appends `Verified: ...` to verified bodies, never raises severity, and lists at most 20 discards. `inlineEligible` no longer posts an AI finding inline once the verifier moved it to the summary. Knobs for experiments: `ALTIMATE_REVIEW_AI_VERIFY_DROP_NITS=0`, `ALTIMATE_REVIEW_AI_VERIFY_KEEP_LOCAL=0`, `ALTIMATE_REVIEW_AI_VERIFY_FAIL=closed`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`runAiVerify` now verifies the AI lane's findings: deterministic features, then one model conversation per batch of at most 8 findings (kept together by file), then triage. It never throws; whatever it could not review comes back unchanged. - Ten read-only tools, declared with `tool()` and no `execute`; the loop runs the calls itself and appends `tool-result` messages: `read_file` (changed files, their dbt parents and direct downstream models only), `compiled_sql`, `model_columns`, `column_reach`, `model_config`, `downstream`, `equivalence`, `grain`, `grep_project`, `pr_description`. Every result is a string of at most 2,500 chars; failures read `not available: ...` - `LLM.stream` runs one step, so the loop drives the rounds: up to 3 tool rounds and 6 calls per round, then a forced answer with `toolChoice: "none"`; the answer is forced early once the batch's share of 20,000 tokens (`ALTIMATE_REVIEW_AI_VERIFY_MAX_TOKENS`) or 16,000 tool-output chars is spent (state `budget`) - one `AbortController` per PR with `aiTimeoutMs` of the prompt size (state `timeout`); a model error or an unparsable answer leaves that batch's findings unreviewed - `harnessVerifierLlm` makes the call through `LLM.stream` at temperature 0 and reports usage with `streamUsage`; tests inject a scripted `VerifierLlm` - status: turns, tool calls, tokens, cost, reviewed, kept, discarded, summary, unreviewed and up to 20 dropped findings Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The verifier's default budget drops from 20,000 to 12,000 tokens per PR (`ALTIMATE_REVIEW_AI_VERIFY_MAX_TOKENS` still overrides it), and the forced final answer now counts against it. Before each call the loop plans the rest of the batch: another tool round is taken only if that call, plus the forced answer after it with a full round of tool output, fits the batch's share. Otherwise it answers now with `toolChoice: "none"`, from the first call on if need be. Call sizes are estimated from the characters sent (system prompt, tool declarations, messages) plus a 1,000-token output reserve, calibrated by the usage the provider reports. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every verifier call repeats the system prompt and the tool declarations, so they are kept to the minimum that stays unambiguous: shorter tool and `side` descriptions, and `read_file` line bounds and the `grep_project` pattern length checked in code rather than in the JSON schema (which emitted `exclusiveMinimum`/`maximum` noise). Tool declarations go from 3,408 to 2,950 chars; the fixed per-call size from 4,832 to 4,374. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`ai.verifier.decisions[]` now records every AI finding the verifier saw, uncapped: its deterministic features, the model's verdict (claim type, verdict, value, assumed input, tools used, and whether it gave evidence; absent when unreviewed), and the decision and rule triage applied. `dropped[]` is unchanged. `retriage(decisions, knobs)` re-runs triage over those records without a model call, so one run's envelopes answer what `dropNits`, `keepUnverifiedLocal` and `failMode` would have changed. It also takes a hypothetical `claimsOtherFileAsCrossFile` knob that sends findings naming another model or source through the cross-file rule; live triage still does not use `claimsOtherFile`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ards `ai.verifier.dropped[]` now carries each discard's `startLine` and body (clamped to 600 chars), so a verifier run can be scored both with and without its discards. That makes the no-verifier baseline a free by-product of the same run instead of a separate one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… budget When a batch's prompt left no room for a tool round, the first call was sent with `toolChoice: "none"` but no instruction to answer. The model then returned about 18 output tokens and no parsable verdicts, so every candidate in the batch went unreviewed. In the decision run this hit 10 PRs (dev and heldout) with large batches. The first message now ends with an explicit answer-now line in that case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the summary The verifier discarded these. In the 50-PR dev decision run, 10 of 21 `unverifiable_cross_file` discards and 9 of 19 `speculative` discards were judged valuable, and every catch the verifier lost came from the first rule. They now stay in the review summary, never inline. Offline re-triage of the same run (no model calls), dev: catches 18 vs 15 with discarding (18 without the verifier); verified AI findings (outside the summary) unchanged at 0.84/PR, 69% valuable, 0% wrong; remaining discards 2 of 18 valuable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…verifier `ALTIMATE_REVIEW_AI_REPLAY=<earlier review --json envelope>` skips the generator call in `runAiReview` and rebuilds its findings from that envelope, so verifier settings can be compared for verifier tokens only. - New `ai-replay.ts`: kept AI findings lose the verifier's `evidence.result.verification`, the `Verified:` body suffix, and the tier/origin set after generation; `ai.verifier.dropped` entries are rebuilt as `Finding`s with the stored `id` and `rule`. Order follows `ai.verifier.decisions`. A `dropped` list shorter than the discard count is warned on stderr and reported as `replayIncomplete`. - The AI lane status copies the stored generator `tokens`/`calls`/`cost` and adds `replayed: true` (envelope schema and `ai_replayed` telemetry). An unreadable envelope is an error on stderr and `state: "error"`; the generator is never called as a fallback. - `ALTIMATE_REVIEW_AI_VERIFY_MAX_TOOL_CHARS` and `ALTIMATE_REVIEW_AI_VERIFY_MAX_ROUNDS` override the verifier budgets; the prompt states the effective round limit. Defaults unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…project false evidence
- `runner.ts`: `columnReach` answers an unknown model (or no manifest) with
`undefined` instead of recording an engine failure, so a verifier tool
argument can no longer land in `engineErrors` and downgrade the verdict.
- `git.ts`: `grepFixedAt` returns `undefined` when git fails or times out with
nothing found, and for an empty pathspec list (which would search the whole
repo); only exit 1 (no match) returns `[]`.
- `ai-verify.ts`: `grep_project` reports a failed search as failed ("not
evidence of absence") rather than "no matches".
- `git.ts`/`run.ts`: new `projectGrepSpecs` builds the search pathspecs; a glob
that already carries the project prefix (the verifier sees repo-relative
paths) is not prefixed again, and a glob with `..` segments is rejected.
- Tests for each.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bcc75a9 to
4733424
Compare
8aba994 to
f3c5e86
Compare
|
Thanks for updating your PR! It now meets our contributing guidelines. 👍 |
| if (glob) { | ||
| const g = glob.trim().replace(/^\.?\/+/, "") | ||
| if (!g || g.split("/").includes("..")) return undefined | ||
| return [root && !g.startsWith(root) ? root + g : g] |
There was a problem hiding this comment.
SUGGESTION: A glob equal to the project root is prefixed a second time
root keeps its trailing slash, so glob: "dbt" (the project root itself, a natural way to ask for the whole project) fails the g.startsWith(root) test and becomes the pathspec dbt/dbt. That directory cannot exist, so the search covers nothing and comes back as no matches/failed instead of searching dbt/ — the same false "evidence of absence" this commit set out to remove. "dbt/" and "dbt/**" work, only the bare root does not.
| return [root && !g.startsWith(root) ? root + g : g] | |
| return [root && g !== root.slice(0, -1) && !g.startsWith(root) ? root + g : g] |
Issue for this PR
Part of #1792
Type of change
What does this PR do?
Stacked on #1758. Adds an opt-in verifier for the AI review lane (
ALTIMATE_REVIEW_AI_VERIFY=1). After the AI reviewer drafts its comments, one more bounded model pass tries to confirm or refute each one from dbt evidence. Comments it can't support are moved to the summary or dropped. Off by default: with the flag unset, the review output is unchanged.Results (replay of public dbt PRs, blind Claude judge)
Verifier on vs off, same runs (50 PRs, 59 reviewer-found defects, tuning repos)
3b2c31e2dd), so catches are not lost.Repos never used for any tuning (25 repos, 101 PRs, one snapshot each, 153 defects)
Difference +11 catches (95% CI +1 to +22, PR-level bootstrap). The verifier costs about 1.4× the tokens of the AI review alone.
What changed
ai-verify.ts:read_file,compiled_sql,model_columns,column_reach,model_config,downstream,equivalence,grain,grep_project,pr_description. At most 3 rounds and 12k tokens.ALTIMATE_REVIEW_AI_REPLAY=<envelope>replays stored generator findings so only the verifier is re-run.columnReach, and the stream reader keeps tool parts.How did you verify your code works?
bun test test/altimate/review: 667 pass.bunx tsc --noEmit: clean.test/altimate/review-ai-verify.test.ts(features, every triage row and knob, tool loop limits, allowlist, fail-open, parser).Cost side effects
None by default: the verifier is opt-in (
ALTIMATE_REVIEW_AI_VERIFY=1). When enabled it adds one bounded model pass per review (at most 3 tool rounds and 12k tokens), measured at about 1.4x the AI-lane tokens of the AI review alone. Its tools read local project files, the manifest and the in-process engine; no warehouse queries. Verifier tokens are recorded in the envelope and telemetry.Screenshots / recordings
Not a UI change.
Checklist
🤖 Generated with Claude Code
Summary by cubic
Adds an opt-in, dbt-grounded verifier for the AI review lane. With
ALTIMATE_REVIEW_AI_VERIFY=1, one bounded model pass after the AI reviewer tries to confirm or refute each comment from dbt evidence (compiled SQL, DAG lineage, project grep); comments it can't support move to the summary or are dropped. With the flag unset the review output is unchanged. The verifier fails open: any error, timeout, or exhausted budget leaves the findings it didn't review untouched.Results (blind judge on public dbt PRs)
On 26 PRs from repos never used for tuning, useful went 38% → 59% and wrong went 1 → 0. Cross-file claims the verifier can't check go to the summary instead of being dropped, so real catches aren't lost. The verifier costs about 1.4× the tokens of the AI review alone.
What changed
retriage()re-runs triage offline without a model call.ALTIMATE_REVIEW_AI_REPLAY=<envelope>replays stored generator findings so verifier settings can be compared without regenerating.grepProjectandcolumnReachare wired into the runner for evidence gathering.Written for commit f3c5e86. Summary will update on new commits.